Many Web 2.0 and Software as a Service (SaaS) applications rely heavily on user created content. This reliance drives the need for (a) efficient and reliable scaling technologies for supporting rapid data growth; and (b) better storage and retrieval technology. Much of this user-created content only requires a primary key for store and retrieve commands rather than complex querying and management functionality offered by traditional Relational Database Management Systems (RDBMS's). The excess RDBMS functionality involves expensive hardware and highly skilled personnel, typically making it unsuitable for these types of applications. In-addition, RDBMS replication capabilities are limited and typically prefer consistency over performance and availability. Despite many developments in recent years, scaling-out a relational database is still very complex.
During recent years NoSQL (Not Only SQL) database management systems (which are also referred to as non-relational databases or unstructured databases) have emerged in-order to solve these RDBMS deficiencies. NoSQL is a broad class of database management systems that can differ from classic RDBMS in some significant ways: (1) there are no inherent relations between stored objects; (2) the data stores may not require fixed table schemas; and (3) NoSQL avoids join operations and typically scales horizontally.
In-memory non-relational databases are subset of NoSQL databases, and are designed in a way that all of (or a major part of) the users dataset is stored in RAM Memory. In-memory non-relational databases are usually in two to three orders of magnitude faster (in terms of throughput and latency) than RDBMS's and an order of magnitude faster than other NoSQL databases.
Among the in-memory non-relational databases, the open source Memcached was first to emerge intending to solve many of the RDBMS issues of read operations, by adding to RDBMS a simple distributed key-value caching system. However, Memcached does not include a data-management layer, and therefore provides no support for high-availability and data-persistence. In addition, during scaling events, Memcached loses all, or significant part of its data.
Redis, an emerging open-source in-memory non-relational database improves Memcached's offering by supporting write operations, persistence storage and high-availability, using a data management-layer for the stored objects. But Redis is built over a single master multi-slave architecture, and therefore suffers from master scaling problems.
Furthermore, due to the relatively high price of Random Access Memory (RAM) resources (as of March 2013, RAM prices are approximately 200 times higher than HDD (Hard Disk Drive)), in-memory non-relational databases may be very expensive. Accordingly, in order to reduce the footprint of in-memory datasets, various techniques that rely on compression and object serialization have been developed. However, such techniques often increase the latency and processing overheads of in-memory non-relational databases beyond what is acceptable. In many cases, sub-millisecond latency is required even at loads exceeding 100,000 requests per second on a single server.
Accordingly, there is a need for new methods, systems, and media for managing an managing in-memory NoSQL database.